Methods and systems for adaptive imaging for low light signal enhancement in medical visualization

ABSTRACT

Adaptive imaging methods and systems for generating enhanced low light video of an object for medical visualization are disclosed and include acquiring, with an image acquisition assembly, a sequence of reference frames and/or a sequence of low light video frames depicting the object, assessing relative movement between the image acquisition assembly and the object based on at least a portion of the acquired sequence of reference video frames or the acquired sequence of low light video frames, adjusting a level of image processing of the low light video frames based at least in part on the relative movement between the image acquisition assembly and the object, and generating a characteristic low light video output from a quantity of the low light video frames, wherein the quantity of the low light video frames is based on the adjusted level of image processing of the low light video frames.

REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application Ser.No. 62/350,121 filed Jun. 14, 2016, titled “ADAPTIVE IMAGING FORFLUORESCENCE SIGNAL ENHANCEMENT,” which is hereby incorporated byreference in its entirety.

FIELD OF THE INVENTION

The present disclosure relates generally to medical imaging. Morespecifically, the disclosure relates to adaptive imaging for low lightsignal enhancement in medical visualization.

BACKGROUND OF THE INVENTION

Imaging technology used in medical visualization (e.g., invasive,minimally-invasive, or non-invasive visualization) can sufferperformance degradation under low light conditions. In particular,fluorescence imaging systems used for medical visualization may need tooperate with very low emitted signal levels from fluorophores at lowconcentration, with limited quantum efficiency and/or deeply embedded inthe tissue.

Low signal levels are not only problematic in cases of fluorescenceimaging, however, but are also known to limit the signal quality ofreflected light laparoscopic images which are acquired through smallaperture optics (e.g. laparoscopes). The combination of low light signalsources (e.g. fluorescence) and small aperture optics (e.g.laparoscopes) may compound the challenge.

Medical imaging systems (e.g., endoscopic imaging systems forminimally-invasive surgery or open field medical imaging systems) canhelp provide clinical information for medical practitioners who need tomake decisions (e.g. intraoperative or treatment decisions) based onvisualization of tissue. In many applications, it is useful for medicalimaging systems to provide white light video in combination with anotherimaging modality (e.g., fluorescence video) substantially simultaneouslyand in real-time. In particular, in applications for visualizing tissue,the white light video is typically acquired by illuminating the tissuewith full visible spectrum light and imaging the illumination light thatis reflected from the tissue surface. In typical applications, suchwhite light video ideally maintains a high color fidelity with the imagethat would be perceived with the normal human eye directly visualizingthe same reflected light. Additionally, fluorescence video, for example,may be acquired by illuminating the tissue with excitation light andimaging the fluorescence light that is emitted by excited fluorophoreslocated in the tissue. The white light video (e.g., color video) and thefluorescence video may be merged and presented to the medicalpractitioner as a single “real-time” video.

There are occasions, however, when there may be a significant disparitybetween the intensity levels of the reflected illumination light and thefluorescence light. In particular, in many instances, the reflectedlight image signal may be orders of magnitude larger than thefluorescence image signal. There are some existing options to compensatefor such image signal differences (e.g., by adjusting the fluorophoreconcentration in the tissue, adjusting the intensity of the excitationlight, amplifying the electronic fluorescence image signal when theoptical fluorescence image signal is transduced at the image sensor orthereafter, etc.). However, these workarounds may prove insufficient foracquiring an adequate fluorescence image signal. In systems that acquirethe reflected light image signal and the fluorescence image signal withthe same image sensor, providing sufficient compensation for arelatively weak fluorescence image signal becomes particularlychallenging.

Furthermore, the reflected light image signal and/or fluorescence imagesignal may suffer from motion blurring as the result of movement of theimaging system and/or the object being imaged. Such motion blurring, aswell as noise in the reflected light and/or fluorescence image signals,may prevent or hamper the visualization of fine details in thefluorescence images.

Thus, it is desirable to have medical imaging systems in whichlow-intensity image signals, such as for example reflected light imagesignals and/or fluorescence image signals, can be more effectivelyvisualized and presented to a user.

SUMMARY OF THE INVENTION

According to some embodiments, an adaptive imaging method for generatinglow light video of an object for medical visualization may includeacquiring, with an image acquisition assembly, a sequence of reflectedlight video frames and/or a sequence of fluorescence video framesdepicting the object, assessing relative movement between the imageacquisition assembly and the object based on reference video frames thatinclude at least a portion of the acquired sequence of reflected lightvideo frames or a portion of the acquired sequence of fluorescence videoframes, adjusting a level of image processing of the reflected lightvideo frames and/or the fluorescence video frames based at least in parton the relative movement (or assessment of relative movement) betweenthe image acquisition assembly and the object, and generating acharacteristic low light video output from a quantity of the reflectedlight video frames and/or a quantity of the fluorescence video frames.In some embodiments, the quantity of the low light video frames for thecharacteristic low light video output may be based on the adjusted levelof image processing of the low light video frames.

Such an adaptive imaging method for use in medical imaging may improvevisualization of low light image signals from an object while limitingthe introduction of image artifacts during relative movement between theimage acquisition assembly and the object. In various embodiments, themethod may be used when acquiring video of a single low light imagesignal (e.g. a reflected white light image signal, or a fluorescenceimage signal) or when acquiring video of a low light image signal whilealso acquiring video of a relatively higher intensity image signal (e.g.a low light fluorescence image signal and a higher intensity reflectedlight image signal). In embodiments that include acquiring video of arelatively higher intensity image signal in addition to a low lightimage signal, the higher intensity image signal may be used as thesource of the reference video frames which may provide for improvedassessment of relative motion between the image acquisition assembly andthe object. In embodiments that include acquiring video of a single lowlight image signal, that low light image signal may be used as thesource of the reference video frames for assessment of relative motion.

In various embodiments, relative movement between the image acquisitionassembly and the object may be assessed by measuring changes in aplurality of the reference video frames, such as change in pixelintensities. For example, change in pixel intensities may be analyzed bydetermining a representative pixel intensity for each of a plurality ofsubregions in the plurality of reference video frames, andcharacterizing the changes in representative pixel intensity for thesubregions in the plurality of the reference video frames.

Following the assessment of relative movement between the imageacquisition assembly and the object, the level of image processing ofthe low light video frames may be adjusted by adjusting the quantity oflow light video frames from which the characteristic low light videooutput is generated. In particular, the image processing level may beadjusted based on at least one motion threshold. For instance, adjustingthe quantity of low light video frames may include setting the quantityof low light video frames to a first predetermined value if the relativemovement (or assessment of relative movement) between the imageacquisition assembly and the object is below a first motion threshold,and setting the quantity of low light video frames to a secondpredetermined value lower than the first predetermined value if therelative movement (or assessment of relative movement) is above thefirst motion threshold. In some variations, additional motion thresholdsmay be utilized, such as by setting the quantity of low light videoframes to the second predetermined value if the relative movement (orassessment of relative movement) is above the first motion threshold andadditionally below a second motion threshold that is higher than thefirst motion threshold. Additionally, the method may include setting thequantity of low light video frames to a third predetermined value thatis lower than the first and second predetermined values if the relativemovement (or assessment of relative movement) is above the first andsecond motion thresholds. In some variations, the quantity of low lightvideo frames may be adjusted or set to a predetermined value bygradually increasing or decreasing the quantity of low light videoframes from which the characteristic low light video output is based,over a series of frames of the characteristic fluorescence video output.

The characteristic low light video output may be generated using variousimage processing steps and based on a quantity of low light video framesassociated with the adjusted level of image processing. For example, aframe of the characteristic low light video output may be generated bydetermining a sum of pixel intensities of the quantity of the low lightvideo frames on a region-by-region basis, and optionally additionallydividing the sum of the pixel intensities by the square root of thequantity of the low light video frames combined to generate the frame ofthe characteristic low light video output. As another example, a frameof the characteristic low light video output may be generated byaveraging pixel intensities of the quantity of low light video frames ona region-by-region basis.

In some variations, the method may additionally or alternatively includeperforming other actions based on the relative movement (or assessmentof relative movement) between the image acquisition assembly and theobject, such as adjusting a low light video frame exposure period and/orcontrolling a timing scheme of the image acquisition assembly, a visiblelight source illuminating the object, and/or an excitation light sourceilluminating the object.

The method may further include displaying the characteristic low lightvideo output on a display. Furthermore, the displaying of the low lightvideo output may be generally continuous, as the method in somevariations may be performed continuously (e.g., as long as the low lightvideo frames are acquired).

Generally, an adaptive imaging system for generating low light video ofan object includes an image acquisition assembly configured to acquire asequence of low light video frames depicting the object, and aprocessor. The processor may be configured to assess relative movementbetween the image acquisition assembly and the object based on a portionof reference video frames, which may include at least a portion of thelow light video frames and/or a portion of substantially simultaneouslyacquired higher intensity light video frames, adjust a level of imageprocessing of the low light video frames based at least in part on therelative movement (or assessment of relative movement) between the imageacquisition assembly and the object, and generate a characteristic lowlight video output from a quantity of the low light video frames,wherein the quantity of the low light video frames is based on theadjusted level of image processing of the low light video frames. Thesystem may further include a visible light source that is configured toemit visible light to illuminate the object, and an excitation lightsource configured to emit excitation light that causes the object toemit fluorescent light. In some variations, the system may furtherinclude a controller that controls a timing scheme for the visible lightsource, the excitation light source, and the image acquisition assemblybased at least in part on the relative movement (or assessment ofrelative movement) between the image acquisition assembly and theobject. The system may include a display that is configured to displaythe characteristic fluorescence video output and/or the reflected lightvideo frames.

According to an embodiment, an adaptive imaging system for generatinglow light fluorescence video of an object includes an image acquisitionassembly configured to acquire a sequence of reflected light videoframes and a sequence of fluorescence video frames depicting the object,and a processor. The processor may be configured to assess relativemovement between the image acquisition assembly and the object based onat least a portion of the reflected light video frames, adjust a levelof image processing of the fluorescence video frames based at least inpart on the relative movement (or assessment of relative movement)between the image acquisition assembly and the object, and generate acharacteristic fluorescence video output from a quantity of thefluorescence video frames, wherein the quantity of the fluorescencevideo frames is based on the adjusted level of image processing of thefluorescence video frames. The system may further include a visiblelight source that is configured to emit visible light to illuminate theobject, and an excitation light source configured to emit excitationlight that causes the object to emit fluorescent light. In somevariations, the system may further include a controller that controls atiming scheme for the visible light source, the excitation light source,and the image acquisition assembly based at least in part on therelative movement (or assessment of relative movement) between the imageacquisition assembly and the object. The system may include a displaythat is configured to display the characteristic fluorescence videooutput and/or the reflected light video frames.

The processor of the adaptive imaging system may be configured to assessrelative movement between the image acquisition assembly and the objectby measuring changes in pixel intensities in a plurality of thereference video frames. Based on this relative movement (or assessmentof relative movement), the processor may be configured to adjust thelevel of image processing of the low light video frames, such as byadjusting the quantity of low light video frames from which thecharacteristic low light video output is generated. More specifically,the processor may set the quantity of low light video frames to a firstpredetermined value if the relative movement (or assessment of relativemovement) between the image acquisition assembly and object is below afirst motion threshold, and set the quantity of low light video framesto a second predetermined value if the relative movement (or assessmentof relative movement) between the image acquisition assembly and objectis above the first motion threshold, wherein the second predeterminedvalue is lower than the first predetermined value. Additionally, theprocessor may set the quantity of low light video frames to the secondpredetermined value if the relative movement (or assessment of relativemovement) between the image acquisition assembly and the object is abovethe first motion threshold and below a second motion threshold higherthan the first motion threshold and set the quantity of low light videoframes to a third predetermined value if the relative movement (orassessment of relative movement) between the image acquisition assemblyand the object is above the first and second motion thresholds, thethird predetermined value being lower than the first and secondpredetermined values. In some variations, the processor may adjust orset the quantity of low light video frames to a predetermined value bygradually increasing or decreasing the quantity of low light videoframes from which the characteristic low light video output is based,over a series of frames of the characteristic low light video output.

The processor may be configured to generate the characteristic low lightvideo output in one or more of various manners, such as by determining asum of pixel intensities of the quantity of the low light video frameson a region-by-region basis and optionally dividing the sum of pixelintensities by the square root of the quantity of low light videoframes, or averaging pixel intensities of the quantity of the low lightvideo frames on a region-by-region basis.

Furthermore, in some variations, some or all of the components of theadaptive imaging system may be combined or integrated with othertechnologies. For example, the adaptive imaging system may furtherinclude an image stabilization system that is implemented in hardware,software, or a combination thereof. As another example, the adaptiveimaging system described herein may be embodied in an endoscopic imagingsystem.

According to some embodiments, an adaptive imaging method for generatingfluorescence video of an object, includes acquiring, with an imageacquisition assembly, a sequence of reflected light video frames and asequence of fluorescence video frames depicting the object, assessingrelative movement between the image acquisition assembly and the objectbased on at least a portion of the acquired sequence of reflected lightvideo frames, adjusting a level of image processing of the fluorescencevideo frames based at least in part on the relative movement between theimage acquisition assembly and the object, and generating acharacteristic fluorescence video output from a quantity of thefluorescence video frames, wherein the quantity of the fluorescencevideo frames is based on the adjusted level of image processing of thefluorescence video frames.

In any of these embodiments, assessing relative movement between theimage acquisition assembly and the object may include measuring changesin pixel intensities in a plurality of the reflected light video frames.In any of these embodiments, assessing relative movement between theimage acquisition assembly and the object may include determining arepresentative pixel intensity for each of a plurality of subregions inthe plurality of reflected light video frames, and characterizing thechanges in representative pixel intensity for the subregions in theplurality of the reflected light video frames.

In any of these embodiments, adjusting the level of image processing ofthe fluorescence video frames may include adjusting the quantity offluorescence video frames from which the characteristic fluorescencevideo output is generated. In any of these embodiments, adjusting thequantity of fluorescence video frames may include setting the quantityof fluorescence video frames to a first predetermined value if therelative movement between the image acquisition assembly and the objectis below a first motion threshold. In any of these embodiments, settingthe quantity of fluorescence video frames to the first predeterminedvalue may include gradually increasing or decreasing the quantity offluorescence video frames to the first predetermined value over a seriesof frames of the characteristic fluorescence video output.

In any of these embodiments, adjusting the quantity of fluorescencevideo frames may include setting the quantity of fluorescence videoframes to a second predetermined value if the relative movement betweenthe image acquisition assembly and the object is above the first motionthreshold, the second predetermined value being lower than the firstpredetermined value. In any of these embodiments, setting the quantityof fluorescence video frames to the second predetermined value mayinclude gradually increasing or decreasing the quantity of fluorescencevideo frames to the second predetermined value over a series of framesof the characteristic fluorescence video output.

In any of these embodiments, adjusting the quantity of fluorescencevideo frames may include setting the quantity of fluorescence videoframes to the second predetermined value if the relative movementbetween the image acquisition assembly and the object is above the firstmotion threshold and below a second motion threshold higher than thefirst motion threshold, and setting the quantity of fluorescence videoframes to a third predetermined value, if the relative movement betweenthe image acquisition assembly and the object is above the first andsecond motion thresholds, the third predetermined value being lower thanthe first and second predetermined values.

In any of these embodiments, setting the quantity of fluorescence videoframes to the third predetermined value may include gradually increasingor decreasing the quantity of fluorescence video frames to the thirdpredetermined value over a series of frames of the characteristicfluorescence video output. In any of these embodiments, adjusting thequantity of fluorescence video frames may include gradually increasingor decreasing the quantity of fluorescence video frames toward apredetermined value over a series of frames of the characteristicfluorescence video output. In any of these embodiments, generating thecharacteristic fluorescence video output may include determining a sumof pixel intensities of the quantity of the fluorescence video frames ona region-by-region basis.

In any of these embodiments, generating the characteristic fluorescencevideo output may further include dividing the sum of pixel intensitiesby the square root of the quantity of the fluorescence video frames. Inany of these embodiments, generating the characteristic fluorescencevideo output from the quantity of the fluorescence video frames mayinclude averaging pixel intensities of the quantity of the fluorescencevideo frames on a region-by-region basis. In any of these embodiments,the method may further include adjusting a fluorescence video frameexposure period based at least in part on the relative movement betweenthe image acquisition assembly and the object.

In any of these embodiments, the method may further include displayingat least one of the characteristic fluorescence video output and thereflected light video frames on a display. In any of these embodiments,the method may further include controlling a timing scheme of a visiblelight source illuminating the object, an excitation light sourceilluminating the object, and the image acquisition assembly based atleast in part on the relative movement between the image acquisitionassembly and the object. In any of these embodiments, the method may beperformed continuously.

According to some embodiments, an adaptive imaging system for generatingfluorescence video of an object, includes an image acquisition assemblyconfigured to acquire a sequence of reflected light video frames and asequence of fluorescence video frames depicting the object, and aprocessor configured to assess relative movement between the imageacquisition assembly and the object based on at least a portion of thereflected light video frames, adjust a level of image processing of thefluorescence video frames based at least in part on the assessedrelative movement between the image acquisition assembly and the object,and generate a characteristic fluorescence video output from a quantityof the fluorescence video frames, wherein the quantity of thefluorescence video frames is based on the adjusted level of imageprocessing of the fluorescence video frames.

In any of these embodiments, the system may further include a visiblelight source configured to emit visible light to illuminate the object,and an excitation light source configured to emit excitation light thatcauses the object to emit fluorescent light. In any of theseembodiments, the system may further include a controller that controls atiming scheme for the visible light source, the excitation light source,and the image acquisition assembly based at least in part on therelative movement between the image acquisition assembly and the object.In any of these embodiments, the processor may be configured to assessrelative movement by measuring changes in pixel intensities in aplurality of the reflected light video frames.

In any of these embodiments, the processor may be configured to adjustthe level of image processing of the fluorescence video frames byadjusting the quantity of fluorescence video frames from which thecharacteristic fluorescence video output is generated. In any of theseembodiments, the processor may be configured to adjust the quantity offluorescence video frames by setting the quantity of fluorescence videoframes to a first predetermined value if the relative movement betweenthe image acquisition assembly and object is below a first motionthreshold, and setting the quantity of fluorescence video frames to asecond predetermined value if the relative movement between the imageacquisition assembly and object is above the first motion threshold, thesecond predetermined value being lower than the first predeterminedvalue.

In any of these embodiments, the processor may be configured to adjustthe quantity of fluorescence video frames by setting the quantity offluorescence video frames to the second predetermined value, if therelative movement between the image acquisition assembly and the objectis above the first motion threshold and below a second motion thresholdhigher than the first motion threshold, and setting the quantity offluorescence video frames to a third predetermined value, if therelative movement between the image acquisition assembly and the objectis above the first and second motion thresholds, the third predeterminedvalue being lower than the first and second predetermined values.

In any of these embodiments, the processor may be configured to adjustthe quantity of fluorescence video frames by gradually increasing ordecreasing the quantity of fluorescence video frames toward apredetermined value over a series of frames of the characteristicfluorescence video output. In any of these embodiments, the processormay be configured to generate the characteristic fluorescence videooutput by performing: (i) determining a sum of pixel intensities of thequantity of the fluorescence video frames on a region-by-region basis,(ii) determining a sum of pixel intensities of the quantity of thefluorescence video frames on a region-by-region basis and dividing thesum of pixel intensities by the square root of the quantity offluorescence video frames, (iii) averaging pixel intensities of thequantity of the fluorescence video frames on a region-by-region basis,or (iv) a combination thereof.

In any of these embodiments, the system may further include a displayconfigured to display at least one of the characteristic fluorescencevideo output and the reflected light video frames. In any of theseembodiments, at least a portion of the imaging system may be embodied inan endoscopic imaging system. In any of these embodiments, the systemmay further include an image stabilization system implemented inhardware, software, or a combination thereof.

According to some embodiments, a computer-implemented, adaptive imagingmethod for generating fluorescence video of an object, includesreceiving a sequence of reflected light video frames and a sequence offluorescence video frames depicting the object, wherein the reflectedlight video frames and fluorescence video frames are acquired by animage acquisition assembly, assessing relative movement between theimage acquisition assembly and the object based on at least a portion ofthe reflected light video frames, adjusting a level of image processingof the fluorescence video frames based at least in part on the relativemovement between the image acquisition assembly and the object, andgenerating a characteristic fluorescence video output from a quantity ofthe fluorescence video frames, wherein the quantity of the fluorescencevideo frames is based on the adjusted level of image processing of thefluorescence video frames.

According to some embodiments, an adaptive imaging method for generatingfluorescence video of an object, includes acquiring, with an imageacquisition assembly, a sequence of reflected light video framesdepicting the object, assessing relative movement between the imageacquisition assembly and the object based on at least a portion of theacquired sequence of reflected light video frames, adjusting afluorescence video frame exposure period based at least in part on therelative movement between the image acquisition assembly and the object,and acquiring a sequence of fluorescence video frames using the adjustedfluorescence video frame exposure period.

According to some embodiments, an adaptive imaging method for generatingenhanced low-intensity light video of an object for medicalvisualization, includes acquiring, with an image acquisition assembly, asequence of low light video frames depicting the object, receiving asequence of reference video frames, assessing relative movement betweenthe image acquisition assembly and the object based on the referencevideo frames, adjusting a level of image processing of the low lightvideo frames based at least in part on the relative movement between theimage acquisition assembly and the object, and generating acharacteristic low light video output from a quantity of the low lightvideo frames, wherein the quantity of the low light video frames isbased on the adjusted level of image processing of the low light videoframes.

According to some embodiments, a kit for imaging an object may include afluorescence imaging agent and the system of any one of the aboveembodiments.

According to some embodiments, a fluorescence imaging agent may includea fluorescence imaging agent for use with the system of any one of theabove embodiments, the method of any one of the above embodiments, orthe kit of any one of the above embodiments.

In any of these embodiments, imaging an object may include imaging anobject during blood flow imaging, tissue perfusion imaging, lymphaticimaging, or a combination thereof.

In any of these embodiments, blood flow imaging, tissue perfusionimaging, and/or lymphatic imaging may include blood flow imaging, tissueperfusion imaging, and/or lymphatic imaging during an invasive surgicalprocedure, a minimally invasive surgical procedure, or during anon-invasive surgical procedure.

In any of these embodiments, the invasive surgical procedure may includea cardiac-related surgical procedure, or a reconstructive surgicalprocedure.

In any of these embodiments, the cardiac-related surgical procedure mayinclude a cardiac coronary artery bypass graft (CABG) procedure.

In any of these embodiments, the CABG procedure may include on pump oroff pump.

In any of these embodiments, the non-invasive surgical procedure mayinclude a wound care procedure.

In any of these embodiments, the lymphatic imaging may includeidentification of a lymph node, lymph node drainage, lymphatic mapping,or a combination thereof.

In any of these embodiments, the lymphatic imaging may relate to thefemale reproductive system.

BRIEF DESCRIPTION OF THE DRAWINGS

Features will become apparent to those of ordinary skill in the art bydescribing in detail exemplary embodiments with reference to theattached drawings in which:

FIG. 1A is an illustrative schematic of an adaptive imaging method forgenerating low light video of an object according to an embodiment; FIG.1B is an illustrative schematic of an adaptive imaging method forgenerating fluorescence video of an object according to an embodiment;

FIG. 2A is an illustrative flowchart of one variation of an adaptiveimaging method for generating low light video of an object according toan embodiment; FIG. 2B is an illustrative flowchart of one variation ofan adaptive imaging method for generating fluorescence video of anobject according to an embodiment;

FIG. 3 is an illustrative schematic of a subsampled image sensor used inone variation of an adaptive imaging method for generating low lightvideo of an object according to an embodiment;

FIG. 4 is a table summarizing exemplary characteristics of imaging modesin one variation of an adaptive imaging method for generating low lightvideo of an object;

FIG. 5 is an illustrative schematic of variations of generating acharacteristic low light video output;

FIG. 6A is an illustrative schematic of one variation of an adaptiveimaging system for generating low light video of an object according toan embodiment; FIG. 6B is an illustrative schematic of one variation ofan adaptive imaging system for generating fluorescence video of anobject according to an embodiment;

FIG. 7 is an illustrative depiction of one variation of an adaptiveimaging system embodied in an endoscopic imaging system according to anembodiment;

FIG. 8 is an illustrative schematic of one variation of a light sourceassembly in an adaptive imaging system according to an embodiment;

FIG. 9 is an illustrative schematic of one variation of an imageacquisition assembly in an adaptive imaging system according to anembodiment; and

FIG. 10 is an illustrative diagram of a range of image quality regimes.

DETAILED DESCRIPTION OF THE INVENTION

Example embodiments will now be described more fully hereinafter withreference to the accompanying drawings; however, they may be embodied indifferent forms and should not be construed as limited to theembodiments set forth herein. Rather, these embodiments are provided sothat this disclosure will be thorough and complete, and will fullyconvey exemplary implementations to those skilled in the art. Variousdevices, systems, methods, processors, kits and imaging agents aredescribed herein. Although at least two variations of the devices,systems, methods, processors, kits and imaging agents are described,other variations may include aspects of the devices, systems, methods,processors, kits and imaging agents described herein combined in anysuitable manner having combinations of all or some of the aspectsdescribed.

Generally, corresponding or similar reference numbers will be used, whenpossible, throughout the drawings to refer to the same or correspondingparts.

Spatially relative terms, such as “beneath”, “below”, “lower”, “above”,“upper”, and the like, may be used herein for ease of description todescribe one element or feature's relationship to another element(s) orfeature(s) as illustrated in the figures. It will be understood that thespatially relative terms are intended to encompass differentorientations of the device in use or operation in addition to theorientation depicted in the figures. For example, if the device in thefigures is turned over, elements described as “below” or “beneath” otherelements or features would then be oriented “above” the other elementsor features. Thus, the exemplary term “below” can encompass both anorientation of above and below. The device may be otherwise oriented(rotated 90 degrees or at other orientations) and the spatially relativedescriptors used herein interpreted accordingly.

In various embodiments, with reference to a particular use environment(e.g., imaging modality, clinical application, or a combinationthereof), “low light video” comprises video wherein the signal to noiseratio (SNR) of the imaged light is relatively low enough such that itmay cause image noise to interfere with the ability to clearly visualizea target feature in the image. In various embodiments, the minimum sizeof a clinically important target feature may vary according to the useenvironment, with a lower minimum target feature size generallyrequiring a higher SNR to yield a given level of feature visibility thana higher minimum target feature size. As shown in FIG. 10, varying videoimage quality regimes with very good visibility of a target feature,moderate visibility, and poor visibility may generally be expected todepend on the SNR, the minimum target feature size for a given useenvironment or a combination thereof. For example, the methods andsystems for adaptive imaging for low light signal enhancement, asdescribed herein in accordance with the various embodiments, mayfacilitate imaging low light video within the moderate visibility orpoor visibility image quality regimes. According to some embodiments,the methods and systems may allow for user input to determine whether toapply the adaptive imaging for a given use environment and ananticipated associated image quality regime. In some variations, theadaptive imaging may be activated or deactivated automatically dependingon the user-indicated use environment.

Generally, the methods and systems described herein may be used togenerate real-time enhanced low light videos (including, for example,reflected light videos and/or fluorescence videos), such as for use inapplications including imaging of tissue (e.g., during endoscopicexaminations, surgical procedures (e.g., minimally invasive), open fieldimaging, and/or other imaging performed with medical imaging systems,including handheld imaging systems). In some embodiments, the low lightvideo may comprise reflected light video that may be based on visiblespectrum light that illuminates and subsequently is reflected fromtissue to be visualized. In some embodiments, the low light video maycomprise fluorescence light video that may be based on fluorescent lightthat is emitted by fluorophores located in the tissue to be visualized,after the fluorophores are excited by excitation spectrum light. In someembodiments, a higher intensity light video may be acquired in additionto the low light video. In particular, the adaptive imaging methods andsystems described herein may be configured to enhance a low light imagesignal such as by compensating for signal noise and/or for motion blur.In some embodiments, the adaptive imaging methods and systems describedherein may be configured to compensate for a low light image signal thatis relatively low-intensity compared to a high-intensity reference imagesignal, and/or compensate for motion blur and/or signal noise.

Adaptive Imaging Method

FIG. 1 illustrates a schematic of an adaptive imaging method for lowlight signal enhancement in medical imaging according to an embodiment.As shown in FIG. 1A, an example of an adaptive imaging method 100 forgenerating fluorescence video of an object may include: acquiring, withan image acquisition assembly, a sequence of low light video depictingthe object 110; assessing relative movement between the imageacquisition assembly and the object 120 based on reference video framescomprising at least a portion of the acquired sequence of low lightvideo frames and/or a portion of a substantially simultaneously acquiredsequence of higher intensity light video frames; adjusting a level ofimage processing of the low light video frames 140 based at least inpart on the relative movement between the image acquisition assembly andthe object; and generating a characteristic low light video output froma quantity of the low light video frames 160, wherein the quantity ofthe low light video frames is based on the adjusted level of imageprocessing of the low light video frames.

In another variation, an adaptive imaging method for generatingfluorescence video of an object may include: acquiring, with an imageacquisition assembly, a sequence of reflected light video frames and asequence of fluorescence video frames depicting the object; assessingrelative movement between the image acquisition assembly and the objectbased on at least a portion of the acquired sequence of reflected lightvideo frames; adjusting a level of image processing of the fluorescencevideo frames based at least in part on the relative movement between theimage acquisition assembly and the object; and generating acharacteristic fluorescence video output from a quantity of thefluorescence video frames, wherein the quantity of the fluorescencevideo frames is based on the adjusted level of image processing of thefluorescence video frames.

In some embodiments, the method may include controlling a timing schemeof one or more light sources and the image acquisition assembly, basedat least in part on the relative movement between the image acquisitionassembly and the object. The method may further include displaying atleast one of the characteristic low light video output (e.g.,fluorescence video output) and the reference video frames 170 (e.g.,reflected light video frames) on a display. In some variations, themethod may be performed continuously throughout acquisition of the lowlight video frames (e.g., reflected light video frames, or fluorescencevideo frames).

In an embodiment, as shown in FIG. 1B, an example of an adaptive imagingmethod 1100 for generating fluorescence video of an object may include:acquiring, with an image acquisition assembly, a sequence of reflectedlight video frames and a sequence of fluorescence video frames depictingthe object 1110; assessing relative movement between the imageacquisition assembly and the object 1120 based on at least a portion ofthe acquired sequence of reflected light video frames; adjusting a levelof image processing of the fluorescence video frames 1140 based at leastin part on the relative movement between the image acquisition assemblyand the object; and generating a characteristic fluorescence videooutput from a quantity of the fluorescence video frames 1160, whereinthe quantity of the fluorescence video frames is based on the adjustedlevel of image processing of the fluorescence video frames. The methodmay include controlling a timing scheme of one or more light sources andthe image acquisition assembly, based at least in part on the relativemovement between the image acquisition assembly and the object. Themethod may further include displaying at least one of the characteristicfluorescence video output and the reflected light video frames 1170 on adisplay. In some variations, the method may be performed continuouslythroughout acquisition of the reflected light video frames andfluorescence video frames.

Acquiring Image Sequences

According to an embodiment, the method may include acquiring a sequenceof low light video frames depicting the object to be visualized (e.g.,tissue), with the use of an image acquisition assembly including atleast one image sensor. Such acquisition may include illuminating theobject with illumination light (e.g., light in the visible lightspectrum) and/or excitation light. The low light video frames maycomprise reflected light video frames that may be obtained with theimage acquisition assembly receiving illumination light that isreflected from the tissue. The reflected light video frames may includecolor images and/or grayscale images, depending on the kind of imagesensors in the image acquisition assembly, as further described below.Additionally, or alternatively, the low light video frames may comprisefluorescence video frames that may be obtained with the imageacquisition assembly receiving fluorescence light that is emitted fromintrinsic and/or extrinsic fluorophores (e.g., a fluorescence imagingagent introduced into the object) that are present in the tissue andexcited by the excitation light. In addition to the sequence of lowlight video frames, the method may include acquiring a sequence ofreference video frames comprising higher light intensity video frames.

According to an embodiment, the method may include acquiring sequencesof reflected light video frames and fluorescence video frames depictingthe object to be visualized (e.g., tissue), with the use of an imageacquisition assembly including at least one image sensor. Suchacquisition may include illuminating the object with illumination light(e.g., light in the visible light spectrum) and excitation light.Reflected light video frames that may be obtained with the imageacquisition assembly receiving illumination light that is reflected fromthe tissue. The reflected light video frames may include color imagesand/or grayscale images, depending on the kind of image sensors in theimage acquisition assembly, as further described below. Fluorescencevideo frames that may be obtained with the image acquisition assemblyreceiving fluorescence light that is emitted from intrinsic and/orextrinsic fluorophores (e.g., a fluorescence imaging agent introducedinto the object) that are present in the tissue and excited by theexcitation light.

In some variations, the low light video frames may be fluorescence videoframes and the reference video frames may be higher light intensityreflected light video frames. The reflected light video frames and thefluorescence video frames may be acquired substantially in parallel orsimultaneously, and in real-time. For example, the reflected light imagesignal and the fluorescence image signal may be acquired with respectiveimage sensors. As another example, in variations in which the samesingle image sensor is used to acquire both reflected light video framesand fluorescence video frames, the acquisition of video frames may beperformed according to a timing scheme. This timing scheme may enableseparation of the image signal associated with the reflected light andthe image signal associated with the fluorescence emission light. Inparticular, the timing scheme may involve illuminating the object withillumination light and excitation light according to a pulsing scheme,and processing the reflected light image signal and fluorescence imagesignal with a processing scheme, wherein the processing scheme issynchronized and matched to the pulsing scheme (e.g., via a controller)to enable separation of the two image signals in a multiplexed manner.Examples of such pulsing and image processing schemes have beendescribed in U.S. Pat. No. 9,173,554, filed on Mar. 18, 2009 and titled“IMAGING SYSTEM FOR COMBINED FULL-COLOR REFLECTANCE AND NEAR-INFRAREDIMAGING,” the contents of which are incorporated in their entirety bythis reference. However, other suitable pulsing and image processingschemes may be used to acquire reflected light video frames andfluorescence video frames simultaneously.

In an embodiment, as the low light video frames and/or the referencevideo frames are acquired, at least a portion of them may be stored(e.g., in a memory unit) for record-keeping purposes and/or retrievalfor analysis during other aspects of the method, as described below.

In an embodiment, as the reflected light video frames and thefluorescence video frames are acquired, at least a portion of them maybe stored (e.g., in a memory unit) for record-keeping purposes and/orretrieval for analysis during other aspects of the method, as describedbelow.

Assessing Relative Movement

According to some embodiments, the method may include assessing therelative movement between the image acquisition assembly and the objectbased on the reference video frames 120. The assessment of such relativemovement may, for example, provide a parameter, where the parameter canbe used to determine the manner in which the low light video framesshould be processed and/or acquired in order to appropriately increasesensitivity or reduce noise in the low light imaging. The relativemovement may result, for example, from unsteady handling of the imageacquisition assembly (e.g., when the image acquisition assembly islocated in a handheld laparoscope or other handheld imaging system), ormovement of a patient being imaged. The relative movement may be betterrepresented in the reference video frames, since the reference videoframes may preferably be acquired in real-time with relatively higherlight intensity and relatively low latency. In some variations, however,a higher light intensity video signal may not be available and thereference video frames may instead comprise frames of the acquired lowlight video frames.

According to some embodiments, the method may include assessing therelative movement between the image acquisition assembly and the objectbased on the reflected light video frames. The assessment of suchrelative movement may, for example, provide a parameter, where theparameter can be used to determine the manner in which the fluorescencevideo frames should be processed and/or acquired in order toappropriately increase sensitivity or reduce noise in the fluorescenceimaging. The relative movement may be better represented in thereflected light video frames, since the reflected light video frames maybe acquired in real-time with relatively higher light intensity andrelatively low latency.

In some variations, assessing relative movement between the imageacquisition assembly and the object 120 may include measuring changes inpixel intensities in a plurality of the reference video frames. Thepixel intensities in the reference video frames may be measured, forexample, from luminance grayscale images based on the reference videoframes. In variations in which the reference video frames are colorimages or white light images (e.g., acquired with one or more imagesensors with a color filter array such as a Bayer pattern filter), themethod may include generating luminance grayscale images from theluminance (brightness) components of the reference video frames. Invariations in which the reference video frames are acquired with anon-color image sensor (e.g., an image sensor without a color filterarray), the reference video frames may need not to be converted into aseparate luminance image.

In some variations, assessing relative movement between the imageacquisition assembly and the object may include measuring changes inpixel intensities in a plurality of the reflected light video frames.The pixel intensities in the reflected light video frames may bemeasured, for example, from luminance grayscale images based on thereflected light video frames. In variations in which the reflected lightvideo frames are color images or white light images (e.g., acquired withone or more image sensors with a color filter array such as a Bayerpattern filter), the method may include generating luminance grayscaleimages from the luminance (brightness) components of the reflected lightvideo frames. In variations in which the reflected light video framesare acquired with a non-color image sensor (e.g., an image sensorwithout a color filter array), the reflected light video frames may neednot to be converted into a separate luminance image.

In an exemplary embodiment shown in FIG. 2A, after acquiring low lightvideo frames (210), assessing the relative movement may includedetermining a representative pixel intensity for one or more subregionsin the reference video frames (222), wherein the reference video framesmay comprise at least a portion of the low light video frames and/or aportion of substantially simultaneously acquired higher intensity lightvideo frames. In particular, each reference video frame may include aplurality of subregions, where each subregion includes a group ofpixels. For example, as shown in FIG. 3, a video frame 340 may includesubregions 342 a and 342 b, where each subregion 342 a or 342 b includesa group of pixels 344 arranged in a cluster (e.g., 1024 pixels arrangedin a 32×32 grid, or 100 pixels arranged in a 10×10 grid as depicted inFIG. 3, etc.). The representative pixel intensity for each of one ormore subregions may be calculated as the average (e.g., mean) intensityof the group of pixels in the subregion, median intensity of the groupof pixels in the subregion, or other manner that is characteristic ofthe overall pixel intensity of the subregion. Utilizing a representativepixel intensity for multi-pixel subregions may reduce computationalcomplexity of the motion assessment and/or reduce the sensitivity to theresolution of the motion assessment. Alternatively, one or more of thesubregions may include a single individual pixel, such as in instancesin which computational complexity is less of a concern or moredetailed/higher resolution intensity information is desired forassessing relative movement.

In an embodiment, after acquiring reflected light and fluorescence videoframes, assessing the relative movement may include determining arepresentative pixel intensity for one or more subregions in thereflected light video frames. In particular, each reflected light videoframe may include a plurality of subregions, where each subregionincludes a group of pixels as described herein.

In an exemplary embodiment shown in FIG. 2B, after acquiring referencereflected light video frames and low light fluorescence video frames(1210), assessing the relative movement may include determining arepresentative pixel intensity for one or more subregions in thereflected light video frames (1222). In particular, each reflected lightvideo frame may include a plurality of subregions, where each subregionincludes a group of pixels. For example, as shown in FIG. 3, a videoframe 340 may include subregions 342 a and 342 b, where each subregion342 a or 342 b includes a group of pixels 344 arranged in a cluster(e.g., 1024 pixels arranged in a 32×32 grid, or 100 pixels arranged in a10×10 grid as depicted in FIG. 3, etc.). The representative pixelintensity for each of one or more subregions may be calculated as theaverage (e.g., mean) intensity of the group of pixels in the subregion,median intensity of the group of pixels in the subregion, or othermanner that is characteristic of the overall pixel intensity of thesubregion. Utilizing a representative pixel intensity for multi-pixelsubregions may reduce computational complexity of the motion assessmentand/or reduce the sensitivity to the resolution of the motionassessment. Alternatively, one or more of the subregions may include asingle individual pixel, such as in instances in which computationalcomplexity is less of a concern or more detailed/higher resolutionintensity information is desired for assessing relative movement.Although FIG. 3 depicts rectangular or grid-like subregions, thesubregions may have any suitable shape. The subregions may besubstantially identical in size and shape, though in some variations,some subregions may be different in size or shape. For example, if someareas of the video frames are identified as more important (e.g., depictan object of interest instead of background) and it is desirable toassess relative motion based on more detailed information for thoseparticularly important areas, subregions in the particularly importantareas may be smaller than less-important regions. All of the subregionsmay be considered in the assessment of relative motion or,alternatively, only a subset of one or more of the subregions may beconsidered, for example to reduce computational complexity and/or tofocus assessment of relative motion on a particular region of interest.For example, in some variations, one of the subsets of alternatingsubregions 342 a and 342 b, may be omitted from consideration in theassessment of relative motion.

In an embodiment, assessing the relative movement may further includecharacterizing the change in the representative pixel intensity for thesubregions in the reference video frames. In an embodiment, assessingthe relative movement may further include characterizing the change inthe representative pixel intensity for the subregions in the reflectedlight video frames. The characterization may be quantitative (e.g., anumerical value describing the magnitude of relative movement betweenthe image acquisition assembly and the object). At least two sequentialvideo frames may be analyzed (e.g., to characterize how therepresentative pixel intensities for the subregions have changed betweenan immediately prior video frame to a current frame, or betweennon-adjacent frames representing endpoints of a multi-frame time periodof interest). For instance, in the exemplary embodiment shown in FIG.2A, assessing the relative movement may further include determining thechange in representative pixel intensity for the subregions 224 anddetermining a statistical measure of the change in representative pixelintensity for the subregions 226. The statistical measure isrepresentative of the assessed relative motion between the imageacquisition assembly and the object. For example, to characterize therelative movement of the image acquisition assembly and object over thecourse of two adjacent reference video frames (or two adjacent reflectedlight video frames), step 224 may include determining the differencebetween the representative pixel intensity for a subregion in a currentvideo frame and the representative pixel intensity for the samesubregion in a previous video frame, and repeating this determinationfor all considered subregions. Subsequently, determining a statisticalmeasure 226 may include, for example, calculating the standard deviationof the differences determined in block 224. As another example,determining a statistical measure 226 may include calculating theaverage (mean, etc.) of the differences determined in block 224. Otherstatistical measures of the change in representative pixel intensity forthe subregions may additionally or alternatively be used to characterizethe change in representative pixel intensity and arrive at a measure ofthe assessed relative motion between the image acquisition assembly andthe object.

In some variations, assessing relative movement between the imageacquisition assembly and the object being imaged may include subsamplingthe pixel intensities of the reflected light video frames. Inparticular, the relative movement may be assessed based on pixelintensities of a selected portion of the subregions (and accordingly, aselected number of the pixels). Such subsampling may reduce the overallcomputational complexity of the motion assessment. For example, as shownin FIG. 3, the relative movement may be based on a subsampling ofsubregions 342 a that are distributed in an alternating checkerboardpattern. In this example, information from subregions 342 a (shaded inFIG. 3) may be utilized in assessing relative motion, while informationfrom subregions 342 b (unshaded in FIG. 3) may be ignored. However, themotion assessment may incorporate any suitable subsampling scheme (e.g.,sampling different portions of the video frames, such as one-third,two-third, or another fraction of the subregions).

The method may additionally or alternatively include any other suitablemotion-estimation algorithms using the pixel intensities and/or othercharacteristics of the reference video frames (e.g., based on a colorcomponent such as red, green, or blue of reference reflected light videoframes, based on chroma values of reference reflected light videoframes, etc.). In some variations, the method may additionally oralternatively include motion-estimation algorithms using the pixelintensities and/or other characteristics of the low light video frames(though, for example, using low light fluorescence video frames insteadof reference reflected light video frames for motion-estimation may beless reliable in some circumstances). Additionally, in some variationsin which the video frames include voxels, the method may includeimplementing the above-described motion-estimation algorithms withrespect to characteristics of voxels (e.g., voxel intensity) instead ofor in addition to characteristics of pixels (e.g., pixel intensity).Furthermore, in some variations the method may additionally oralternatively include receiving information from a gyroscope or otherhardware configured to detect motion of the image acquisition assembly,and utilizing this information to assess movement of the imageacquisition assembly.

The method may additionally or alternatively include any other suitablemotion-estimation algorithms using the pixel intensities and/or othercharacteristics of the reflected light video frames (e.g., based on acolor component such as red, green, or blue of reflected light videoframes, based on chroma values of the reflected light video frames,etc.). In some variations, the method may additionally or alternativelyinclude motion-estimation algorithms using the pixel intensities and/orother characteristics of the fluorescence frames (though, for example,using fluorescence video frames instead of the reflected light videoframes for motion-estimation may be less reliable in somecircumstances).

Between iterations, one or more of the values calculated during motionassessment may be stored in memory for future use. For example, therepresentative pixel intensities for the subregions in a current videoframe may be stored for use in a future iteration of the calculations,to be used as the representative pixel intensities for the subregions ina previous video frame.

Adjusting the Level of Image Processing

Generally, the method may include selecting an imaging mode suitable fordifferent amounts of relative movement between the image acquisitionassembly and the object being imaged. In some variations, upon selectionof an imaging mode, the method includes adjusting a level of imageprocessing of the low light video frames 140 (e.g., fluorescence videoframes) based on the relative movement (or assessment of relativemovement) between the image acquisition assembly and the object beingimaged. The level of image processing may be adjusted in order toincrease sensitivity and/or reduce noise in the characteristicfluorescence video output.

For instance, upon selection of an imaging mode, adjusting the level ofimage processing of the low light video frames may include adjusting thequantity of low light video frames from which the characteristic lowlight video output is generated. At least some of the imaging modes maycorrespond to a respective, predetermined plural quantity of low lightvideo frames that may be combined (as described in further detail below)to generate a characteristic low light video output with amplified orenhanced low light data from the combined low light video frames.However, more motion artifacts (e.g., motion blur) may appear in thecharacteristic low light video output if there is a significant amountof relative movement between the image acquisition assembly and theobject throughout the combined low light video frames. Thus, in order toreduce motion artifacts in the characteristic low light video output,the quantity of low light video frames that is used to generate eachframe of the characteristic low light video output may generally beinversely related to the degree of relative movement between the imageacquisition assembly and the object being imaged.

For example, as shown in FIG. 4, a “low-motion mode” may correlate to arelatively high quantity of low light video frames (e.g., fluorescencevideo frames) that are combined to generate a frame of the low lightvideo output during the low-motion mode. This low-motion mode may besuitable, for instance, when there are no or negligible changes inrelative positions of the image acquisition assembly and the objectbeing imaged. A “moderate-motion mode” may correlate to a moderatequantity of low light video frames that are combined to generate a frameof the low light video output during the moderate-motion mode. Thismoderate-motion mode may be suitable, for instance, when there are smallor slight changes in relative positions of the image acquisitionassembly and the object being imaged. A “high-motion mode” may correlateto a relatively low quantity (e.g., one) of low light video frames thatare combined to generate a frame of the low light video output duringthe high-motion mode. This high-motion mode may be suitable, forinstance, when there are major changes in relative positions of theimage acquisition assembly and the object being imaged, and when it isdesirable to avoid smearing or motion blur of the fluorescence imagefeatures and/or desirable to reduce the time lag in the displayedfluorescence video output.

Although FIG. 4 depicts an exemplary embodiment of the method with threeimaging modes, other embodiments of the method may have fewer (e.g., 2)or more (e.g., 4, 5, or 6, etc.) imaging modes following a similar trendwith respect to quantity of low light video frames (e.g., fluorescencevideo frames) that are combined to generate the characteristic low lightvideo output (e.g., fluorescence video output).

Referring to FIG. 2A, after a measure representative of the assessedrelative movement between the image acquisition assembly and the objectis determined (226), the measure may be compared to one or more motionthresholds. Based on these comparisons, the particular imaging mode maybe set to low-motion mode (252), moderate-motion mode (254), orhigh-motion mode (256). Subsequently, adjusting the level of imageprocessing of the low light video frames (e.g., fluorescence videoframes) may include adjusting the quantity of low light video framesfrom which the characteristic low light video output is generated,according to the set imaging mode.

More specifically, if the measure representative of the assessedrelative movement is below a low-motion threshold (242), then the methodmay set the imaging system to a low-motion mode (252) such thatadjusting the quantity of low light video frames involves setting thequantity of low light video frames (from which the characteristic lowlight video output is generated) to a predetermined high value.Additionally, if the measure is above the low-motion threshold (242),then the measure may be compared to a high-motion threshold (244). Ifthe measure is above the low-motion threshold but below the high-motionthreshold (244), then the method may set the imaging system to amoderate-motion mode (254) such that adjusting the quantity of low lightvideo frames involves setting the quantity of low light video frames toa predetermined moderate value that is lower than the predetermined highvalue. If the measure is above the high-motion threshold (244), then themethod may set the imaging system to a high-motion mode (256) such thatadjusting the quantity of low light video frames involves setting thequantity of low light video frames (from which the characteristic lowlight video output is generated) to a predetermined low value that islower than the predetermined moderate and high values.

In some variations, when the imaging system is set to a new imagingmode, the method may include gradually increasing or decreasing thequantity of low light video frames (from which the characteristic lowlight video output is generated) to the predetermined value associatedwith the new imaging mode. For instance, the gradual increasing ordecreasing may occur over a series of frames of the characteristic lowlight video output, as defined by a predetermined number of frames or apredetermined period of time. This gradual change may help provide asmooth visual transition between previously-selected andcurrently-selected imaging modes. For example, when transitioning to thelow-motion mode, setting the quantity of low light video frames to thepredetermined high value may include gradually increasing the quantityof low light video frames. Similarly, when transitioning from thelow-motion or the high-motion imaging mode to the moderate-motion mode,setting the quantity of low light video frames to the predetermined midvalue may include gradually decreasing or increasing, respectively, thequantity of low light video frames. When transitioning to thehigh-motion mode, setting the quantity of low light video frames to thepredetermined low value may include gradually decreasing the quantity oflow light video frames in some variations. However, in some variations,it may be particularly desirable to immediately set the quantity to thepredetermined low value when transitioning to the high-motion mode, toreduce motion artifacts in the low light video output when they are mostlikely to occur.

Generating a Characteristic Image

The method may include generating a characteristic low light videooutput 160 (e.g., a fluorescence video output) from the quantity of lowlight video frames (e.g., fluorescence video frames) based on theadjusted level of image processing. Generally, the quantity of low lightvideo frames may be combined into a frame of the characteristic lowlight video output such that the low light video output visualizesamplified and/or de-noised low light image data from the combined lowlight video frames.

As shown in FIG. 5, given a quantity N of low light video frames 550(e.g., fluorescence video frames), the N frames may be combined in anyone or more of several manners to generate a particular frame of thecharacteristic low light video output. In one variation, generating thecharacteristic low light video output 560 may include determining a sumof pixel intensities of the N low light video frames 550 on aregion-by-region basis 562. Summing the pixel intensities may increasethe low light signal intensity in the characteristic low light videooutput.

In another variation, generating the characteristic low light videooutput 560 (e.g., fluorescence video output) may include determining asum of pixel intensities of the N low light video frames 550 on aregion-by-region basis 564 (similar to 562) and further dividing the sumof the pixel intensities by a scaling factor 566, such as the squareroot of the number of video frames N. Other suitable scaling factorsbesides solely the square root of N may additionally or alternatively beused to scale the sum of the pixel intensities. This variation maypartially increase the low light signal intensity in the characteristiclow light video output, while limiting or eliminating any correspondingincrease in the intensity of the image noise.

In yet another variation, generating the characteristic low light videooutput 560 may include averaging the pixel intensities of the quantity Nof low light video frames 568. This may preserve the low light signalintensity in the characteristic low light video output, but mayadditionally reduce the intensity of image noise (that is, increase thesignal-to-noise ratio).

In other variations, the characteristic low light video output may begenerated from a selected quantity of low light video frames 550 inother suitable manners.

Other Low Light Imaging Adjustments

In another variation, at least some of the imaging modes mayadditionally or alternatively correspond to a respective low light videoframe (e.g., a respective fluorescence video frame) exposure period.Longer exposure periods allow for an amplified low light image signal,but are also associated with a higher risk of motion artifacts resultingfrom relative movement between the image acquisition assembly and theobject throughout the exposure period. Thus, in order to reduce motionartifacts in the characteristic low light video output, the duration ofthe exposure period may generally be inversely related to the degree ofrelative movement between the image acquisition assembly and the objectbeing imaged. For example, as shown in FIG. 4, a low-motion mode imagingmay be associated with a relatively high exposure (i.e., relativelylonger period of time), a moderate-motion imaging mode may be associatedwith a moderate exposure, and a high-motion imaging mode may beassociated with a relatively low exposure (i.e., relatively shorterperiod of time). Although FIG. 4 depicts an exemplary embodiment of themethod with three imaging modes, other embodiments of the method mayhave fewer (e.g., 2) or more (e.g., 4, 5, or 6, etc.) imaging modesfollowing a similar trend with respect to the low light video frameexposure period.

The adjustment of low light video frame exposure period may besupplemental to the multi-frame low light image processing describedabove for the different imaging modes. Alternatively, the adjustment oflow light video frame exposure period may be performed without themulti-frame low light image processing, such that the method includesadjusting a low light video frame exposure period based at least in parton the relative movement between the image acquisition assembly and theobject being imaged, and acquiring a sequence of low light video framesusing the adjusted low light video frame exposure period.

Additionally, similar to the above-described gradual transition in thequantity of low light video frames being used for generating thecharacteristic low light video output, the transition between theimaging modes may involve gradually lengthening or shortening the lowlight video frame exposure period over a series of frames of thecharacteristic low light video output.

In other variations, such as for variations in which reference lightvideo frames (e.g., reflected light video frames) and low light videoframes (e.g., fluorescence video frames) are acquired with a singleimage sensor, the method may further include controlling a timing schemeof a visible light source that is illuminating the object, an excitationlight source that is illuminating the object, and the image acquisitionassembly. The timing scheme may be controlled based at least in part onthe relative movement between the image acquisition assembly and theobject. For instance, in one example, the timing scheme in amoderate-motion imaging mode may involve a repeated pattern ofillumination and image processing for two successive reflective lightvideo frames followed by one fluorescence light video frame. In alow-motion mode, the timing scheme may involve a repeated pattern ofillumination and image processing for one reflective light video framefollowed by two fluorescence light video frames which facilitatesfluorescence image signal amplification and/or noise reduction in thecharacteristic fluorescence video output. Similar adjustments to thenumber of frames for fluorescence image acquisition may be performed forother variations of timing sequences.

Another example may be implemented in instances where two colorcomponent signals (e.g., green and blue) are continuously read on two ofthree channels on an image sensor while a third color component signal(e.g., red) and a fluorescence signal are alternatively read on thethird channel of the image sensor. In this example, the timing scheme ina moderate-motion imaging mode may involve a repeated pattern of theillumination and image processing for two successive video frames withthe third color component and one video frame with the fluorescence(e.g., GB+R, GB+R, GB+FL). In a low-motion imaging mode, the timingscheme may involve a repeated pattern of illumination and imageprocessing for one video frame with the third color component and twosuccessive video frames with the fluorescence (e.g., GB+R, GB+FL, GB+FL)which facilitates fluorescence image signal amplification and/or noisereduction in the characteristic fluorescence video output. Similaradjustments to the number of frames for fluorescence image acquisitionmay be performed for other variations of timing sequences.

In yet another variation, the method may include applying imagestabilization technology to compensate for at least some ranges ofmotion or relative movement between the image acquisition assembly andthe object (e.g., due to unsteady handling of the image acquisitionassembly). The image stabilization technology may, for example, beimplemented in hardware and/or image processing software, either as partof the imaging system or as a separate plug-in electronic imagestabilization module. For instance, the level of image processing of thelow light video frames (e.g., fluorescence video frames) may be adjustedbased on an assessment of the residual evidence of relative movementthat remains in at least a portion of the acquired sequence of referencevideo frames (e.g., reflected light video frames) after compensation bythe image stabilization technology (e.g., instead of based on theactual, greater amount of relative movement determinable from videoframes without compensation by the image stabilization technology).

Displaying

As shown in FIG. 1A, the method may include displaying at least one ofthe characteristic low light video output (e.g., fluorescence videooutput) and the reference video frames 170 (e.g., reflected light videoframes) on a display (e.g., monitor or screen). In some instances (e.g.,based on operator-selected settings), the method may include displayingboth the characteristic low light video output and the sequence ofreference video frames, either side-by-side or overlaid or otherwisemerged. If the characteristic low light video output and the sequence ofreference video frames are merged, then the low light video output maybe displayed in high contrast to the reference video frames such as in adisplay color that is not commonly present in the body (e.g., brightgreen, purple). Similarly, the method may include displaying low lightvideo frames acquired using an adjusted low light video frame exposureperiod, and/or the reference video frames either side-by-side or mergedas described above.

The method may include storing or printing at least some frames of thecharacteristic low light video output, acquired low light video frames,and/or acquired reference video frames. For instance, desired videoframes may be selected by the user via a user interface for storing in amemory unit, display, printing, etc.

Adaptive Imaging System

Generally, as shown in FIG. 6A, an example of an adaptive imaging system600 for generating low light video of an object 602 (e.g., a tissueregion of interest) may include: an image acquisition assembly 620 withat least one image sensor 622 configured to acquire a sequence of lowlight video frames depicting the object; and a processor 630. Theprocessor 630 is configured to assess relative movement between theimage acquisition assembly 620 based on reference video framescomprising at least a portion of the low light video frames and/or aportion of a substantially simultaneously acquired sequence of higherintensity video frames, adjust a level of processing of the low lightvideo frames based at least in part on the relative movement between theimage acquisition assembly 620 and the object 602, and generate acharacteristic low light video output from a quantity of the low lightvideo frames, wherein the quantity of the low light video frames isbased on the adjusted level of image processing of the low light videoframes. In other variations, the processor 630 may be configured toperform aspects of the method 100 described above.

In accordance with some embodiments, as shown in FIG. 6B, an example ofan adaptive imaging system 1600 for generating fluorescence video of anobject 1602 (e.g., a tissue region of interest) may include: an imageacquisition assembly 1620 with at least one image sensor 1622 configuredto acquire a sequence of reflected light video frames and a sequence offluorescence video frames depicting the object; and a processor 1630.The processor 1630 is configured to assess relative movement between theimage acquisition assembly 1620 based on at least a portion of thereflected light video frames, adjust a level of processing of thefluorescence video frames based at least in part on the relativemovement between the image acquisition assembly 1620 and the object1602, and generate a characteristic fluorescence video output from aquantity of the fluorescence video frames, wherein the quantity of thefluorescence video frames is based on the adjusted level of imageprocessing of the fluorescence video frames. In other variations, theprocessor 1630 may be configured to perform aspects of the method 1100described above.

In some variations, at least part of the adaptive imaging system may beembodied in an endoscopic imaging system, such as for minimally-invasiveprocedures. For example, as shown in FIG. 7, an endoscopic imagingsystem 700 may include an illuminator 702 with a light source assemblyconfigured to provide visible light and/or fluorescence excitation lightto a surgical laparoscope 704 via a light guide 706 that is connected tothe illuminator 702 via a light guide port 708. A processor 710 and/orcontroller 720 may, in some variations, be within the same housing asthe illuminator 702, as shown in FIG. 7, and may be configured toperform at least some of the aspects of the method 100 described above.An image acquisition assembly 712 may receive signals via connection tothe laparoscope 704, and may pass acquired images to the processor 710via connection to the processor 710 such as through port 714. Certainaspects of the light source assembly, image acquisition assembly,processor, and/or controller may be similar to those described in moredetail below.

Light Source Assembly

As shown in the schematic of FIG. 6A, the imaging system 600 may includea light source assembly 610 including a visible light source 612 thatemits visible light (e.g., full spectrum visible light, narrow bandvisible light, or other portions of the visible light spectrum) and/oran excitation light source 614 that emits excitation light for excitingfluorophores in the object 602 and causing fluorescence emission.

The visible light source 612 is configured to emit visible light forillumination of the object to be imaged. In some variations, the visiblelight source may include one or more solid state emitters, such as LEDsand/or laser diodes. For example, the visible light source may includeblue, green, and red (or other color components) LEDs or laser diodesthat in combination generate white light illumination. These colorcomponent light sources may be centered around the same wavelengthsaround which the image acquisition assembly (described further below) iscentered. For example, in variations in which the image acquisitionassembly includes a single chip, single color image sensor having an RGBcolor filter array deposited on its pixels, the red, green, and bluelight sources may be centered around the same wavelengths around whichthe RGB color filter array is centered. As another example, invariations in which the image acquisition assembly includes athree-chip, three-sensor (RGB) color camera system, the red, green, andblue light sources may be centered around the same wavelengths aroundwhich the red, green, and blue image sensors are centered.

The excitation light source 614 is configured to emit excitation lightsuitable for exciting intrinsic fluorophores and/or extrinsicfluorophores (e.g., a fluorescence imaging agent introduced into theobject) located in the object being imaged. The excitation light source614 may include, for example, one or more LEDs, laser diodes, arc lamps,and/or illuminating technologies of sufficient intensity and appropriatewavelength to excite the fluorophores located in the object beingimaged. For example, the excitation light source may be configured toemit light in the near-infrared (NIR) waveband (such as, for example,approximately 805 nm light), though other excitation light wavelengthsmay be appropriate depending on the application.

The light source assembly 610 may further include one or more opticalelements that shape and/or guide the light output from the visible lightsource 612 and/or excitation light source 614. The optical componentsmay include one or more lenses, mirrors (e.g., dichroic mirrors), lightguides and/or diffractive elements, e.g., so as to help ensure a flatfield over substantially the entire field of view of the imageacquisition assembly 620. For example, as shown in the schematic of FIG.8, the output 824 from a laser diode 822 (providing visible light orexcitation light) may be passed through one or more focusing lenses 826,and then through a light guide 828. The light may be further passedthrough an optical diffractive element 832 (e.g., one or more opticaldiffusers). Power to the laser diode 822 may be provided by, forexample, a high-current laser driver and may optionally be operated in apulsed mode during the image acquisition process according to a timingscheme. An optical sensor such as a solid state photodiode 830 may beincorporated into the light source assembly and may sample theillumination intensity produced by one or more of the light sources, viascattered or diffuse reflections from the various optical elements.

Image Acquisition Assembly

The image acquisition assembly 620 may acquire reflected light videoframes based on visible light that has reflected from the object, and/orfluorescence video frames based on fluorescence emitted by fluorophoresin the object that are excited by the fluorescence excitation light. Asshown in FIG. 9, the image acquisition assembly 620 may acquire imagesusing a system of optics (e.g., one or more lenses 946 a, one or morefilters 948, one or more mirrors 950, beam splitters, etc.) to collectand focus reflected light and/or fluorescent light 942 onto an imagesensor assembly 944. The image sensor assembly 944 may include at leastone solid state image sensor. The one or more image sensors may include,for example, a charge coupled device (CCD), a CMOS sensor, a CID, orother suitable sensor technology. In one variation, the image sensorassembly 944 may include a single chip, single image sensor (e.g., agrayscale image sensor or a color image sensor having an RGB colorfilter array deposited on its pixels). In another variation, the imageacquisition assembly may include a three-chip, three-sensor (RGB) imagesensor assembly 944.

Processor and Controller

As shown in the schematic of FIG. 6A, the system may include a processor630. The processor may include, for example, a microprocessor or othersuitable central processing unit. In particular, the processor 630 maybe configured to execute instructions to perform aspects of the methodsdescribed herein. As the low light video frames and/or the referencevideo frames are acquired, at least a portion of them may be stored in amemory unit for record-keeping purposes and/or retrieval for analysisduring other aspects of the method, as described below.

As shown in the schematic of FIG. 6A, the system may include acontroller 640, which may be embodied in, for example, a microprocessorand/or timing electronics. In some variations, a single image sensor maybe used to acquire both low light video frames and reference videoframes, and the controller 640 may control a timing scheme for thevisible light source and/or the excitation light source, and the imageacquisition assembly. This timing scheme may enable separation of theimage signal associated with the low light signal and the image signalassociated with the higher intensity reference light signal. Inparticular, the timing scheme may involve illuminating the object withillumination light and/or excitation light according to a pulsingscheme, and processing the low light image signal and reference imagesignal with a processing scheme, wherein the processing scheme issynchronized and matched to the pulsing scheme (e.g., via a controller)to enable separation of the two image signals in a time-divisionmultiplexed manner. Examples of such pulsing and image processingschemes have been described in U.S. Pat. No. 9,173,554, filed on Mar.18, 2009 and titled “IMAGING SYSTEM FOR COMBINED FULL-COLOR REFLECTANCEAND NEAR-INFRARED IMAGING,” the contents of which are incorporated intheir entirety by this reference. However, other suitable pulsing andimage processing schemes may be used to acquire reference video framesand low light video frames simultaneously, for example to acquirereflected light video frames and fluorescence video framessimultaneously. Furthermore, the controller may be configured to controlthe timing scheme for the visible light source and/or the excitationlight source, and the image acquisition assembly based at least in parton the relative movement between the image acquisition assembly and theobject.

Other Hardware

In some variations, the system may include image stabilizing technologythat helps compensate for some ranges of motion (e.g., caused byunsteady hands holding the image acquisition assembly) in the acquiredlow light images and/or reference images. The image stabilizingtechnology may be implemented in hardware, such as with optical imagestabilization technology that counteracts some relative movement betweenthe image acquisition assembly and the object by varying the opticalpath to the image sensor (e.g., lens-based adjustments and/orsensor-based adjustments). Additionally, or alternatively, the imagestabilization technology may be implemented in software, such as withdigital image stabilization that counteracts some relative movementbetween the image acquisition assembly and the object (e.g., by shiftingthe electronic image between video frames, utilizing stabilizationfilters with pixel tracking, etc.). Such image stabilizing technologymay, for example, help correct for motion blur in the characteristic lowlight video output (or in the acquired low light video frames) resultingfrom relative motion during long exposure periods or the combination ofmultiple low light video frames.

The system may, in some variations, include one or more hardware motionsensors (e.g., gyroscope, accelerometer) that measure absolute motion ofthe image acquisition assembly. Information from these motion-measuringsensors may be used, in addition or as an alternative to theabove-described motion-estimation algorithms, to determine which imagingmode of the system is suitable for a given set of circumstances.

Additionally, the system may include one or more data modules 640 thatcommunicates and/or stores some or all of the acquired reference videoframes, acquired low light video frames, characteristic low light videooutput, and/or information generated from the image data. For instance,the data module 640 may include a display (e.g., computer screen orother monitor), recorder or other data storage device, printer, and/orpicture archiving and communication system (PACS). The system mayadditionally or alternatively include any suitable systems forcommunicating and/or storing images and image-related data.

A kit may include any part of the systems described herein, and/or thetangible non-transitory computer-readable medium described above havingcomputer-executable (readable) program code embedded thereon that mayprovide instructions for causing one or more processors, when executingthe instructions, to perform one or more of the methods describedherein. For instance, the instructions may cause one or more processors,when executing the instructions, to perform an adaptive imaging methodfor generating low light video of an object. The method comprisesreceiving a sequence of low light video frames depicting the object,wherein the low light video frames are acquired by an image acquisitionassembly; assessing relative movement between the image acquisitionassembly and the object based on reference video frames comprising atleast a portion of the low light video frames and/or a portion of asubstantially simultaneously acquired sequence of higher light intensityvideo frames; adjusting a level of image processing of the low lightvideo frames based at least in part on the relative movement between theimage acquisition assembly and the object; and generating acharacteristic low light video output from a quantity of low light videoframes, wherein the quantity of the low light video frames is based onthe adjusted level of image processing of the low light video frames.Furthermore, the kit may include instructions for use of at least someof its components (e.g., for installing the computer-executable(readable) program code with instructions embedded thereon, etc.).

In other variations, a kit may include any part of the systems describedherein and a fluorescence agent such as, for example, a fluorescence dyesuch as ICG or any suitable fluorescence agent or a combination offluorescence agents. In some variations, a suitable fluorescence agentis an agent which can circulate with the blood (e.g., an agent which cancirculate with, for example, a component of the blood such as plasma inthe blood) and which fluoresces when exposed to appropriate excitationlight energy. ICG, when administered to the subject, binds with bloodproteins and circulates with the blood in the tissue. The fluorescenceimaging agent (e.g., ICG) may be administered to the subject as a bolusinjection (e.g., into a vein or an artery) in a concentration suitablefor imaging such that the bolus circulates in the vasculature andtraverses the microvasculature. In other embodiments in which multiplefluorescence imaging agents are used, such agents may be administeredsimultaneously, e.g. in a single bolus, or sequentially in separateboluses. In some embodiments, the fluorescence imaging agent may beadministered by a catheter. In certain embodiments, the fluorescenceimaging agent may be administered less than an hour in advance ofperforming the measurement of signal intensity arising from thefluorescence imaging agent. For example, the fluorescence imaging agentmay be administered to the subject less than 30 minutes in advance ofthe measurement. In yet other embodiments, the fluorescence imagingagent may be administered at least 30 seconds in advance of performingthe measurement. In still other embodiments, the fluorescence imagingagent may be administered contemporaneously with performing themeasurement. According to some embodiments, the fluorescence imagingagent may be administered in various concentrations to achieve a desiredcirculating concentration in the blood. For example, in embodimentswhere the fluorescence imaging agent is ICG, it may be administered at aconcentration of about 2.5 mg/mL to achieve a circulating concentrationof about 5 μM to about 10 μM in blood. In various embodiments, the upperconcentration limit for the administration of the fluorescence imagingagent is the concentration at which the fluorescence imaging agentbecomes clinically toxic in circulating blood, and the lowerconcentration limit is the instrumental limit for acquiring the signalintensity data arising from the fluorescence imaging agent circulatingwith blood to detect the fluorescence imaging agent. In various otherembodiments, the upper concentration limit for the administration of thefluorescence imaging agent is the concentration at which thefluorescence imaging agent becomes self-quenching. For example, thecirculating concentration of ICG may range from about 2 μM to about 10mM. Thus, in one aspect, the method comprises the step of administrationof the imaging agent (e.g., a fluorescence imaging agent) to the subjectand acquisition of the signal intensity data (e.g., video) prior toprocessing the signal intensity data according to the variousembodiments. In another aspect, the method excludes any step ofadministering the imaging agent to the subject.

According to some embodiments, a suitable fluorescence imaging agent foruse in fluorescence imaging applications to generate fluorescence imagedata is an imaging agent which can circulate with the blood (e.g., afluorescence dye which can circulate with, for example, a component ofthe blood such as lipoproteins or serum plasma in the blood) and transitvasculature of the tissue (i.e., large vessels and microvasculature),and from which a signal intensity arises when the imaging agent isexposed to appropriate light energy (e.g., excitation light energy, orabsorption light energy). In various embodiments, the fluorescenceimaging agent comprises a fluorescence dye, an analogue thereof, aderivative thereof, or a combination of these. An example of thefluorescence agent is a fluorescence dye, which includes any non-toxicfluorescence dye. In certain variations, the fluorescence dye mayinclude a dye that emits light in the near-infrared spectrum. In certainembodiments, the fluorescence dye may include a tricarbocyanine dye suchas, for example, indocyanine green (ICG). In other variations, thefluorescence dye may comprise methylene blue, ICG or a combinationthereof. In certain embodiments the dye is or comprises fluoresceinisothiocyanate, rhodamine, phycoerythrin, phycocyanin, allophycocyanin,o-phthaldehyde, fluorescamine, rose Bengal, trypan blue, fluoro-gold,green fluorescence protein, flavins (e.g., riboflavin, etc.), methyleneblue, porphysomes, cyanine dyes (e.g., cathepsin-activated Cy5 combinedwith a targeting ligand, Cy5.5, etc.), IRDye800CW, CLR 1502 combinedwith a targeting ligand, OTL38 combined with a targeting ligand, or acombination thereof, which is excitable using excitation lightwavelengths appropriate to each imaging agent. In some variations, ananalogue or a derivative of the fluorescence imaging agent may be used.For example, a fluorescence dye analogue or a derivative may include afluorescence dye that has been chemically modified, but still retainsits ability to fluoresce when exposed to light energy of an appropriatewavelength. In variations in which some or all of the fluorescence isderived from autofluorescence, one or more of the fluorophores givingrise to the autofluorescence may be an endogenous tissue fluorophore(e.g., collagen, elastin, NADH, etc.), 5-aminolevulinic acid (5-ALA), ora combination thereof.

In various embodiments, the fluorescence imaging agent may be providedas a lyophilized powder, solid, or liquid. In certain embodiments, thefluorescence imaging agent may be provided in a vial (e.g., a sterilevial), which may permit reconstitution to a suitable concentration byadministering a sterile fluid with a sterile syringe. Reconstitution maybe performed using any appropriate carrier or diluent. For example, thefluorescence imaging agent may be reconstituted with an aqueous diluentimmediately before administration. In various embodiments, any diluentor carrier which will maintain the fluorescence imaging agent insolution may be used. As an example, ICG may be reconstituted withwater. In some embodiments, once the fluorescence imaging agent isreconstituted, it may be mixed with additional diluents and carriers. Insome embodiments, the fluorescence imaging agent may be conjugated toanother molecule, such as a protein, a peptide, an amino acid, asynthetic polymer, or a sugar, for example to enhance solubility,stability, imaging properties, or a combination thereof. Additionalbuffering agents may optionally be added including Tris, HCl, NaOH,phosphate buffer, and/or HEPES.

A person of skill in the art will appreciate that, although afluorescence imaging agent was described above in detail, other imagingagents may be used in connection with the systems, methods, andtechniques described herein, depending on the medical imaging modality.

In some variations, the fluorescence imaging agent used in combinationwith the methods, systems and kits described herein may be used forblood flow imaging, tissue perfusion imaging, lymphatic imaging, or acombination thereof, which may be performed during an invasive surgicalprocedure, a minimally invasive surgical procedure, a non-invasivesurgical procedure, or a combination thereof. Examples of invasivesurgical procedure which may involve blood flow and tissue perfusioninclude a cardiac-related surgical procedure (e.g., CABG on pump or offpump) or a reconstructive surgical procedure. An example of anon-invasive or minimally invasive procedure includes wound (e.g.,chronic wound such as for example pressure ulcers) treatment and/ormanagement. In this regard, for example, a change in the wound overtime, such as a change in wound dimensions (e.g., diameter, area), or achange in tissue perfusion in the wound and/or around the peri-wound,may be tracked over time with the application of the methods andsystems. Examples of lymphatic imaging include identification of one ormore lymph nodes, lymph node drainage, lymphatic mapping, or acombination thereof. In some variations, such lymphatic imaging mayrelate to the female reproductive system (e.g., uterus, cervix, vulva).

In variations relating to cardiac applications or any vascularapplications, the imaging agent(s) (e.g., ICG alone or in combinationwith another imaging agent) may be injected intravenously. For example,the imaging agent may be injected intravenously through the centralvenous line, bypass pump and/or cardioplegia line and/or othervasculature to flow and/or perfuse the coronary vasculature,microvasculature and/or grafts. ICG may be administered as a diluteICG/blood/saline solution down the grafted vessel or other vasculaturesuch that the final concentration of ICG in the coronary artery or othervasculature depending on application is approximately the same or loweras would result from injection of about 2.5 mg (i.e., 1 ml of 2.5 mg/ml)into the central line or the bypass pump. The ICG may be prepared bydissolving, for example, 25 mg of the solid in 10 ml sterile aqueoussolvent, which may be provided with the ICG by the manufacturer. Onemilliliter of the ICG solution may be mixed with 500 ml of sterilesaline (e.g., by injecting 1 ml of ICG into a 500 ml bag of saline).Thirty milliliters of the dilute ICG/saline solution may be added to 10ml of the subject's blood, which may be obtained in an aseptic mannerfrom the central arterial line or the bypass pump. ICG in blood binds toplasma proteins and facilitates preventing leakage out of the bloodvessels. Mixing of ICG with blood may be performed using standardsterile techniques within the sterile surgical field. Ten ml of theICG/saline/blood mixture may be administered for each graft. Rather thanadministering ICG by injection through the wall of the graft using aneedle, ICG may be administered by means of a syringe attached to the(open) proximal end of the graft. When the graft is harvested surgeonsroutinely attach an adaptor to the proximal end of the graft so thatthey can attach a saline filled syringe, seal off the distal end of thegraft and inject saline down the graft, pressurizing the graft and thusassessing the integrity of the conduit (with respect to leaks, sidebranches etc.) prior to performing the first anastomosis. In othervariations, the methods, dosages or a combination thereof as describedherein in connection with cardiac imaging may be used in any vascularand/or tissue perfusion imaging applications.

Lymphatic mapping is an important part of effective surgical staging forcancers that spread through the lymphatic system (e.g., breast, gastric,gynecological cancers). Excision of multiple nodes from a particularnode basin can lead to serious complications, including acute or chroniclymphedema, paresthesia, and/or seroma formation, when in fact, if thesentinel node is negative for metastasis, the surrounding nodes willmost likely also be negative. Identification of the tumor draining lymphnodes (LN) has become an important step for staging cancers that spreadthrough the lymphatic system in breast cancer surgery for example. LNmapping involves the use of dyes and/or radiotracers to identify the LNseither for biopsy or resection and subsequent pathological assessmentfor metastasis. The goal of lymphadenectomy at the time of surgicalstaging is to identify and remove the LNs that are at high risk forlocal spread of the cancer. Sentinel lymph node (SLN) mapping hasemerged as an effective surgical strategy in the treatment of breastcancer. It is generally based on the concept that metastasis (spread ofcancer to the axillary LNs), if present, should be located in the SLN,which is defined in the art as the first LN or group of nodes to whichcancer cells are most likely to spread from a primary tumor. If the SLNis negative for metastasis, then the surrounding secondary and tertiaryLN should also be negative. The primary benefit of SLN mapping is toreduce the number of subjects who receive traditional partial orcomplete lymphadenectomy and thus reduce the number of subjects whosuffer from the associated morbidities such as lymphedema andlymphocysts.

The current standard of care for SLN mapping involves injection of atracer that identifies the lymphatic drainage pathway from the primarytumor. The tracers used may be radioisotopes (e.g. Technetium-99 orTc-99m) for intraoperative localization with a gamma probe. Theradioactive tracer technique (known as scintigraphy) is limited tohospitals with access to radioisotopes require involvement of a nuclearphysician and does not provide real-time visual guidance. A colored dye,isosulfan blue, has also been used, however this dye cannot be seenthrough skin and fatty tissue. In addition, blue staining results intattooing of the breast lasting several months, skin necrosis can occurwith subdermal injections, and allergic reactions with rare anaphylaxishave also been reported. Severe anaphylactic reactions have occurredafter injection of isosulfan blue (approximately 2% of patients).Manifestations include respiratory distress, shock, angioedema,urticarial and pruritus. Reactions are more likely to occur in subjectswith a history of bronchial asthma, or subjects with allergies or drugreactions to triphenylmethane dyes. Isosulfan blue is known to interferewith measurements of oxygen saturation by pulse oximetry andmethemoglobin by gas analyzer. The use of isosulfan blue may result intransient or long-term (tattooing) blue coloration.

In contrast, fluorescence imaging in accordance with the variousembodiments for use in SLN visualization, mapping, facilitates directreal-time visual identification of a LN and/or the afferent lymphaticchannel intraoperatively, facilitates high-resolution optical guidancein real-time through skin and fatty tissue, visualization of blood flow,tissue perfusion or a combination thereof.

In some variations, visualization, classification or both of lymph nodesduring fluorescence imaging may be based on imaging of one or moreimaging agents, which may be further based on visualization and/orclassification with a gamma probe (e.g., Technetium Tc-99m is a clear,colorless aqueous solution and is typically injected into theperiareolar area as per standard care), another conventionally usedcolored imaging agent (isosulfan blue), and/or other assessment such as,for example, histology. The breast of a subject may be injected, forexample, twice with about 1% isosulfan blue (for comparison purposes)and twice with an ICG solution having a concentration of about 2.5mg/ml. The injection of isosulfan blue may precede the injection of ICGor vice versa. For example, using a TB syringe and a 30 G needle, thesubject under anesthesia may be injected with 0.4 ml (0.2 ml at eachsite) of isosulfan blue in the periareolar area of the breast. For theright breast, the subject may be injected at 12 and 9 o'clock positionsand for the left breast at 12 and 3 o'clock positions. The total dose ofintradermal injection of isosulfan blue into each breast may be about4.0 mg (0.4 ml of 1% solution: 10 mg/ml). In another exemplaryvariation, the subject may receive an ICG injection first followed byisosulfan blue (for comparison). One 25 mg vial of ICG may bereconstituted with 10 ml sterile water for injection to yield a 2.5mg/ml solution immediately prior to ICG administration. Using a TBsyringe and a 30G needle, for example, the subject may be injected withabout 0.1 ml of ICG (0.05 ml at each site) in the periareolar area ofthe breast (for the right breast, the injection may be performed at 12and 9 o'clock positions and for the left breast at 12 and 3 o'clockpositions). The total dose of intradermal injection of ICG into eachbreast may be about 0.25 mg (0.1 ml of 2.5 mg/ml solution) per breast.ICG may be injected, for example, at a rate of 5 to 10 seconds perinjection. When ICG is injected intradermally, the protein bindingproperties of ICG cause it to be rapidly taken up by the lymph and movedthrough the conducting vessels to the LN. In some variations, the ICGmay be provided in the form of a sterile lyophilized powder containing25 mg ICG with no more than 5% sodium iodide. The ICG may be packagedwith aqueous solvent consisting of sterile water for injection, which isused to reconstitute the ICG. In some variations the ICG dose (mg) inbreast cancer sentinel lymphatic mapping may range from about 0.5 mg toabout 10 mg depending on the route of administration. In somevariations, the ICG does may be about 0.6 mg to about 0.75 mg, about0.75 mg to about 5 mg, about 5 mg to about 10 mg. The route ofadministration may be for example subdermal, intradermal (e.g., into theperiareolar region), subareolar, skin overlaying the tumor, intradermalin the areola closest to tumor, subdermal into areola, intradermal abovethe tumor, periareolar over the whole breast, or a combination thereof.The NIR fluorescent positive LNs (e.g., using ICG) may be represented asa black and white NIR fluorescence image(s) for example and/or as a fullor partial color (white light) image, full or partial desaturated whitelight image, an enhanced colored image, an overlay (e.g., fluorescencewith any other image), a composite image (e.g., fluorescenceincorporated into another image) which may have various colors, variouslevels of desaturation or various ranges of a color tohighlight/visualize certain features of interest. Processing of theimages may be further performed for further visualization and/or otheranalysis (e.g., quantification). The lymph nodes and lymphatic vesselsmay be visualized (e.g., intraoperatively, in real time) usingfluorescence imaging systems and methods according to the variousembodiments for ICG and SLNs alone or in combination with a gamma probe(Tc-99m) according to American Society of Breast Surgeons (ASBrS)practice guidelines for SLN biopsy in breast cancer patients.Fluorescence imaging for LNs may begin from the site of injection bytracing the lymphatic channels leading to the LNs in the axilla. Oncethe visual images of LNs are identified, LN mapping and identificationof LNs may be done through incised skin, LN mapping may be performeduntil ICG visualized nodes are identified. For comparison, mapping withisosulfan blue may be performed until ‘blue’ nodes are identified. LNsidentified with ICG alone or in combination with another imagingtechnique (e.g., isosulfan blue, and/or Tc-99m) may be labeled to beexcised. Subject may have various stages of breast cancer (e.g., IA, IB,IIA).

In some variations, such as for example, in gynecological cancers (e.g.,uterine, endometrial, vulvar and cervical malignancies), ICG may beadministered interstitially for the visualization of lymph nodes,lymphatic channels, or a combination thereof. When injectedinterstitially, the protein binding properties of ICG cause it to berapidly taken up by the lymph and moved through the conducting vesselsto the SLN. ICG may be provided for injection in the form of a sterilelyophilized powder containing 25 mg ICG (e.g., 25 mg/vial) with no morethan 5.0% sodium iodide. ICG may be then reconstituted with commerciallyavailable water (sterile) for injection prior to use. According to anembodiment, a vial containing 25 mg ICG may be reconstituted in 20 ml ofwater for injection, resulting in a 1.25 mg/ml solution. A total of 4 mlof this 1.25 mg/ml solution is to be injected into a subject (4×1 mlinjections) for a total dose of ICG of 5 mg per subject. The cervix mayalso be injected four (4) times with a 1 ml solution of 1% isosulfanblue 10 mg/ml (for comparison purposes) for a total dose of 40 mg. Theinjection may be performed while the subject is under anesthesia in theoperating room. In some variations the ICG dose (mg) in gynecologicalcancer sentinel lymph node detection and/or mapping may range from about0.1 mg to about 5 mg depending on the route of administration. In somevariations, the ICG does may be about 0.1 mg to about 0.75 mg, about0.75 mg to about 1.5 mg, about 1.5 mg to about 2.5 mg, about 2.5 mg toabout 5 mg. The route of administration may be for example cervicalinjection, vulva peritumoral injection, hysteroscopic endometrialinjection, or a combination thereof. In order to minimize the spillageof isosulfan blue or ICG interfering with the mapping procedure when LNsare to be excised, mapping may be performed on a hemi-pelvis, andmapping with both isosulfan blue and ICG may be performed prior to theexcision of any LNs. LN mapping for Clinical Stage I endometrial cancermay be performed according to the NCCN Guidelines for Uterine Neoplasms,SLN Algorithm for Surgical Staging of Endometrial Cancer; and SLNmapping for Clinical Stage I cervical cancer may be performed accordingto the NCCN Guidelines for Cervical Neoplasms, Surgical/SLN MappingAlgorithm for Early-Stage Cervical Cancer. Identification of LNs maythus be based on ICG fluorescence imaging alone or in combination orco-administration with for a colorimetric dye (isosulfan blue) and/orradiotracer.

Visualization of lymph nodes may be qualitative and/or quantitative.Such visualization may comprise, for example, lymph node detection,detection rate, anatomic distribution of lymph nodes. Visualization oflymph nodes according to the various embodiments may be used alone or incombination with other variables (e.g., vital signs, height, weight,demographics, surgical predictive factors, relevant medical history andunderlying conditions, histological visualization and/or assessment,Tc-99m visualization and/or assessment, concomitant medications).Follow-up visits may occur on the date of discharge, and subsequentdates (e.g., one month).

Lymph fluid comprises high levels of protein, thus ICG can bind toendogenous proteins when entering the lymphatic system. Fluorescenceimaging (e.g., ICG imaging) for lymphatic mapping when used inaccordance with the methods and systems described herein offers thefollowing example advantages: high-signal to background ratio (or tumorto background ratio) as NIR does not generate significantautofluorescence, real-time visualization feature for lymphatic mapping,tissue definition (i.e., structural visualization), rapid excretion andelimination after entering the vascular system, and avoidance ofnon-ionizing radiation. Furthermore, NIR imaging has superior tissuepenetration (approximately 5 to 10 millimeters of tissue) to that ofvisible light (1 to 3 mm of tissue). The use of ICG for example alsofacilitates visualization through the peritoneum overlying thepara-aortic nodes. Although tissue fluorescence can be observed with NIRlight for extended periods, it cannot be seen with visible light andconsequently does not impact pathologic evaluation or processing of theLN. Also, florescence is easier to detect intra-operatively than bluestaining (isosulfan blue) of lymph nodes. In other variations, themethods, dosages or a combination thereof as described herein inconnection with lymphatic imaging may be used in any vascular and/ortissue perfusion imaging applications.

Tissue perfusion relates to the microcirculatory flow of blood per unittissue volume in which oxygen and nutrients are provided to and waste isremoved from the capillary bed of the tissue being perfused. Tissueperfusion is a phenomenon related to but also distinct from blood flowin vessels. Quantified blood flow through blood vessels may be expressedin terms that define flow (i.e., volume/time), or that define speed(i.e., distance/time). Tissue blood perfusion defines movement of bloodthrough micro-vasculature, such as arterioles, capillaries, or venules,within a tissue volume. Quantified tissue blood perfusion may beexpressed in terms of blood flow through tissue volume, namely, that ofblood volume/time/tissue volume (or tissue mass). Perfusion isassociated with nutritive blood vessels (e.g., micro-vessels known ascapillaries) that comprise the vessels associated with exchange ofmetabolites between blood and tissue, rather than larger-diameternon-nutritive vessels. In some embodiments, quantification of a targettissue may include calculating or determining a parameter or an amountrelated to the target tissue, such as a rate, size volume, time,distance/time, and/or volume/time, and/or an amount of change as itrelates to any one or more of the preceding parameters or amounts.However, compared to blood movement through the larger diameter bloodvessels, blood movement through individual capillaries can be highlyerratic, principally due to vasomotion, wherein spontaneous oscillationin blood vessel tone manifests as pulsation in erythrocyte movement.

One or more embodiments are directed to a fluorescence imaging agent foruse in the imaging systems and methods as described herein. In one ormore embodiments, the use may comprise blood flow imaging, tissueperfusion imaging, lymphatic imaging, or a combination thereof, whichmay occur during an invasive surgical procedure, a minimally invasivesurgical procedure, a non-invasive surgical procedure, or a combinationthereof. The fluorescence agent may be included in the kit describedherein.

In one or more embodiments, the invasive surgical procedure may comprisea cardiac-related surgical procedure or a reconstructive surgicalprocedure. The cardiac-related surgical procedure may comprise a cardiaccoronary artery bypass graft (CABG) procedure which may be on pumpand/or off pump.

In one or more embodiments, the minimally invasive or the non-invasivesurgical procedure may comprise a wound care procedure.

In one or more embodiments, the lymphatic imaging may compriseidentification of a lymph node, lymph node drainage, lymphatic mapping,or a combination thereof. The lymphatic imaging may relate to the femalereproductive system.

The methods and processes described herein may be performed by code orinstructions to be executed by a computer, processor, manager, orcontroller, or in hardware or other circuitry. Because the algorithmsthat form the basis of the methods (or operations of the computer,processor, or controller) are described in detail, the code orinstructions for implementing the operations of the method embodimentsmay transform the computer, processor, or controller into aspecial-purpose processor for performing the methods described herein.

Also, another embodiment may include a computer-readable medium, e.g., anon-transitory computer-readable medium, for storing the code orinstructions described above. The computer-readable medium may be avolatile or non-volatile memory or other storage device, which may beremovably or fixedly coupled to the computer, processor, or controllerwhich is to execute the code or instructions for performing the methodembodiments described herein.

Example embodiments have been disclosed herein, and although specificterms are employed, they are used and are to be interpreted in a genericand descriptive sense only and not for purpose of limitation. In someinstances, as would be apparent to one of ordinary skill in the art asof the filing of the present application, features, characteristics,and/or elements described in connection with a particular embodiment maybe used singly or in combination with features, characteristics, and/orelements described in connection with other embodiments unless otherwisespecifically indicated. Accordingly, it will be understood by those ofskill in the art that various changes in form and details may be madewithout departing from the spirit and scope of the present invention asset forth in the following.

While the present disclosure has been illustrated and described inconnection with various embodiments shown and described in detail, it isnot intended to be limited to the details shown, since variousmodifications and structural changes may be made without departing inany way from the scope of the present disclosure. Various modificationsof form, arrangement of components, steps, details and order ofoperations of the embodiments illustrated, as well as other embodimentsof the disclosure may be made without departing in any way from thescope of the present disclosure, and will be apparent to a person ofskill in the art upon reference to this description. It is thereforecontemplated that the appended claims will cover such modifications andembodiments as they fall within the true scope of the disclosure. Forthe purpose of clarity and a concise description features are describedherein as part of the same or separate embodiments, however, it will beappreciated that the scope of the disclosure includes embodiments havingcombinations of all or some of the features described. For the terms“for example” and “such as,” and grammatical equivalences thereof, thephrase “and without limitation” is understood to follow unlessexplicitly stated otherwise. As used herein, the singular forms “a”,“an”, and “the” include plural referents unless the context clearlydictates otherwise.

What is claimed is:
 1. An adaptive imaging method for generatingenhanced low-intensity light video of an object for medicalvisualization, comprising: acquiring, with an image acquisitionassembly, a sequence of low light video frames depicting the object;receiving a sequence of reference video frames; assessing relativemovement between the image acquisition assembly and the object based onthe reference video frames; adjusting a level of image processing of thelow light video frames based at least in part on the relative movementbetween the image acquisition assembly and the object; and generating acharacteristic low light video output from a quantity of the low lightvideo frames, wherein the quantity of the low light video frames isbased on the adjusted level of image processing of the low light videoframes.
 2. The method of claim 1, wherein the sequence of referencevideo frames comprises frames from the sequence of low light videoframes.
 3. The method of claim 1, further comprising acquiring, with theimage acquisition assembly, a sequence of higher intensity light videoframes generally having a higher light intensity than the low lightvideo frames, wherein the sequence of higher intensity light videoframes are acquired substantially simultaneously with the sequence oflow light video frames.
 4. The method of claim 3, wherein the sequenceof reference video frames comprises frames from the sequence of higherintensity light video frames.
 5. The method of claim 1, whereinassessing relative movement between the image acquisition assembly andthe object comprises measuring changes in pixel intensities in aplurality of the reference video frames.
 6. The method of claim 1,wherein assessing relative movement between the image acquisitionassembly and the object comprises determining a representative pixelintensity for each of a plurality of subregions in the plurality ofreference video frames, and characterizing the changes in representativepixel intensity for the subregions in the plurality of the referencevideo frames.
 7. The method of claim 1, wherein adjusting the level ofimage processing of the low light video frames comprises adjusting thequantity of low light video frames from which the characteristic lowlight video output is generated.
 8. The method of claim 1, whereinadjusting the quantity of low light video frames comprises setting thequantity of low light video frames to a first predetermined value if therelative movement between the image acquisition assembly and the objectis below a first motion threshold.
 9. The method of claim 8, whereinsetting the quantity of low light video frames to the firstpredetermined value comprises gradually increasing or decreasing thequantity of low light video frames to the first predetermined value overa series of frames of the characteristic low light video output.
 10. Themethod of claim 9, wherein adjusting the quantity of low light videoframes comprises setting the quantity of low light video frames to asecond predetermined value if the relative movement between the imageacquisition assembly and the object is above the first motion threshold,the second predetermined value being lower than the first predeterminedvalue.
 11. The method of claim 10, wherein setting the quantity of lowlight video frames to the second predetermined value comprises graduallyincreasing or decreasing the quantity of low light video frames to thesecond predetermined value over a series of frames of the characteristiclow light video output.
 12. The method of claim 11, wherein adjustingthe quantity of low light video frames comprises: setting the quantityof low light video frames to the second predetermined value if therelative movement between the image acquisition assembly and the objectis above the first motion threshold and below a second motion thresholdhigher than the first motion threshold; and setting the quantity of lowlight video frames to a third predetermined value, if the relativemovement between the image acquisition assembly and the object is abovethe first and second motion thresholds, the third predetermined valuebeing lower than the first and second predetermined values.
 13. Themethod of claim 12, wherein setting the quantity of low light videoframes to the third predetermined value comprises gradually increasingor decreasing the quantity of low light video frames to the thirdpredetermined value over a series of frames of the characteristic lowlight video output.
 14. The method of claim 13, wherein adjusting thequantity of low light video frames comprises gradually increasing ordecreasing the quantity of low light video frames toward a predeterminedvalue over a series of frames of the characteristic low light videooutput.
 15. The method of claim 1, wherein generating the characteristiclow light video output comprises determining a sum of pixel intensitiesof the quantity of the low light video frames on a region-by-regionbasis.
 16. The method of claim 15, wherein generating the characteristiclow light video output further comprises dividing the sum of pixelintensities by the square root of the quantity of the low light videoframes.
 17. The method of claim 1, wherein generating the characteristiclow light video output from the quantity of the low light video framescomprises averaging pixel intensities of the quantity of the low lightvideo frames on a region-by-region basis.
 18. The method of claim 17,further comprising adjusting a low light video frame exposure periodbased at least in part on the relative movement between the imageacquisition assembly and the object.
 19. The method of claim 1, furthercomprising displaying at least one of the characteristic low light videooutput and the reference video frames on a display.
 20. The method ofclaim 1, further comprising controlling a timing scheme of a visiblelight source illuminating the object, an excitation light sourceilluminating the object, and the image acquisition assembly based atleast in part on the relative movement between the image acquisitionassembly and the object.
 21. The method of claim 1, wherein the methodis performed continuously.
 22. The method of claim 1, wherein thesequence of low light video frames comprises a sequence of fluorescencevideo frames.
 23. The method of claim 1, wherein the sequence of lowlight video frames comprises a sequence of reflected light video frames.24. The method of claim 1, wherein the sequence of reference videoframes comprises a sequence of reflected light video frames.
 25. Anadaptive imaging system for generating enhanced low-intensity lightvideo of an object for medical visualization, comprising: an imageacquisition assembly configured to acquire a sequence of low light videoframes depicting the object; one or more processors; memory; and one ormore programs, wherein the one or more programs are stored in the memoryand configured to be executed by the one or more processors, the one ormore programs including instructions for: assessing relative movementbetween the image acquisition assembly and the object based on referencevideo frames; adjusting a level of image processing of the low lightvideo frames based at least in part on the relative movement between theimage acquisition assembly and the object; and generating acharacteristic low light video output from a quantity of the low lightvideo frames, wherein the quantity of the low light video frames isbased on the adjusted level of image processing of the low light videoframes.
 26. A fluorescence imaging agent for use with the method ofclaim 1 for imaging an object during blood flow imaging, tissueperfusion imaging, lymphatic imaging, or a combination thereof.